Definition 9.1 A point estimate is any function T (X 1,..., X n ) of a random sample. We often write an estimator of the parameter θ as ˆθ.

Transcription

1 9 Point estimation 9.1 Rationale behind point estimation When sampling from a population described by a pdf f(x θ) or probability function P [X = x θ] knowledge of θ gives knowledge of the entire population. Hence it is natural to seek a method of finding a good estimator of θ. Often the parameter θ has a meaningful physical interpretation, so there is a direct interest in obtaining a good estimate of θ. It may be the case that some function of θ, say τ(θ), is of interest. Definition 9.1 A point estimate is any function T (X 1,..., X n ) of a random sample. We often write an estimator of the parameter θ as ˆθ. An estimator of θ is a function of random variables, so is itself a random variable. A value of the estimator for any realization x 1,..., x n of the random sample, that is T (x 1,..., x n ) is a real number and is called an estimate. 9.2 Properties of good estimators There are a number of properties of estimators which may be desirable. 1. Unbiasedness. If E[ˆθ] = θ we say the estimator is unbiased. It means that the distribution of the random variable ˆθ should be centred about the true value θ. The bias of an estimator is defined as Bias(ˆθ) = E[ˆθ] θ. 2. Small variance. The variance of the estimator should be small. estimator about its mean should be small. The spread of the Var[ˆθ] = E[(ˆθ E(ˆθ)) 2 ] 1

2 3. Small mean square error. If the estimator ˆθ is biased then it makes more sense to want the spread about θ to be small. This is measured by the mean square error. MSE(ˆθ) = E[(ˆθ θ) 2 ] Theorem 9.1 The mean square error can be expressed as the sum of the variance of the estimator and the square of the bias, MSE(ˆθ) = Var[ˆθ] + (Bias(ˆθ)) 2. The identity is easily obtained MSE(ˆθ) = E[(ˆθ θ) 2 ] = E[(ˆθ E(ˆθ) + E(ˆθ) θ) 2 ] = E[(ˆθ E(ˆθ)) 2 ] + E[(E(ˆθ) θ) 2 ] + 2 E[(ˆθ E(ˆθ))(E(ˆθ) θ)] = E[(ˆθ E(ˆθ)) 2 ] + (E(ˆθ) θ) 2 = Var[ˆθ] + (Bias(ˆθ)) 2 since the cross product term is zero because E[(ˆθ E(ˆθ))] = 0 and the square of the bias is a constant. 4. Consistency Consistency is a large-sample property, since it describes the limiting behaviour of the estimator ˆθ as the sample size tends to infinity. Definition 9.2 If ˆθ n is an estimator of θ based on a random sample of size n, we say that ˆθ n is consistent for θ if lim P ( ˆθ n θ < ɛ) = 1. n A sufficient condition for consistency is that lim MSE(ˆθ n ) = 0. n 2

3 The MSE is an important criterion for comparing two estimators. Let ˆθ 1 and ˆθ 2 be two estimators of the parameter θ. Then the relative efficiency of ˆθ 1 to ˆθ 2 is defined as Eff = MSE(ˆθ 2 ) MSE(ˆθ 1 ). If Eff < 1 we would conclude that ˆθ 2 is a better estimator than ˆθ 1. Note that if ˆθ 1 and ˆθ 2 are unbiased then Eff = Var(ˆθ 2 ) Var(ˆθ 1 ). Example Let X 1,..., X n be a random sample from a normal population with mean µ and variance σ 2. One obvious estimator of the mean µ is the sample mean X. We know that E[ X] = µ and Var[ X] = σ 2 /n. Consider the alternative estimator ˆθ = X 1. We know that E[X 1 ] = µ and Var[X 1 ] = σ 2. Thus the efficiency of the estimator X 1 to the sample mean is Eff = MSE( X) MSE(X 1 ) = σ2 /n = 1 σ 2 n. Hence the sample mean is a better estimator than a single observation. Note that since MSE( X) = σ 2 /n 0 n we see that X is a consistent estimator of µ. 9.3 Methods of estimation Method of moments Suppose that X is a continuous rv with pdf f(x θ 1,..., θ k ) or a discrete rv with probability function P (X = x θ 1,..., θ k ) characterised by k unknown parameters. Let X 1,..., X n be a random sample from X. The first k sample moments about the origin are defined as m t = 1 n Xi t t = 1, 2,..., k. 3

4 The first k population moments about the origin are µ t = E[X t ] = x t f(x θ 1,..., θ k )dx if X is continuous, or µ t = E[X t ] = j x t jp (X = x j θ 1,..., θ k ) if X is discrete. The population moments are, in general, functions of the k unknown parameters θ t. Equating sample moments and population moments yields k simultaneous equations in k unknowns, θ t, t = 1,..., k, that is µ t = m t t = 1,..., k. The solution to the system, denoted by ˆθ 1,..., ˆθ k gives the moment estimators of the parameters θ 1,..., θ k. Example Let X N(µ, σ 2 ) where µ and σ 2 are unknown. Let X 1,..., X n be a random sample from X. The population moments are The sample moments are µ 1 = E[X] = µ µ 2 = E[X 2 ] = σ 2 + µ 2. m 1 = 1 n m 2 = 1 n X i Xi 2. Hence we have the system of two equations µ = 1 n σ 2 + µ 2 = 1 n X i = X Xi 2 4

5 which have the solution ˆµ = X ˆσ 2 = 1 n Xi 2 X 2 = 1 n ( X 2 i n X 2 ) = 1 n (Xi X) 2. So the moment estimator of µ is the sample mean. The estimator of variance is the sample second moment about the sample mean. It is not the sample variance (and hence it is biased). Example Let X 1,..., X n be a random sample from a population having a uniform distribution on the interval (a, b), where a and b are unknown. Use the method of moments to find estimators of a and b. Let the following values be a realisation of a random sample of size 10: 2.3, 4.2, 5.3, 5.7, 8.1, 2.8, 6.2, 4.4, 8.5, 3.5. Calculate the moment estimates of a and b based on these data. Do the estimates have reasonable values? Let X i U(a, b), i = 1,..., n. The moment equations are: { X = E(X) = µ 1 n n X2 i = E(X 2 ) = σ 2 + µ 2 { µ = X σ 2 = 1 n n X2 i µ 2 Also, for a uniform distribution U(a, b) we have E(X) = a+b and var(x) = 2 (b a) 2. Hence, 12 { a+b = X 2 (b a) 2 = 1 n 12 n X2 i µ 2 { a + b = 2 X b a = 12( 1 n n X2 i µ2 ). 5

6 Hence, { a = 2 X b 2b = 2 X + 12( 1 n n X2 i µ2 ) Finally, â = X 3( 1 n n X2 i X 2 ) ˆb = X + 3( 1 n n X2 i X 2 ) The estimates of a and of b for the given data are: â = = 1.67 ˆb = = The estimates seem to be reasonable, as â < min x i and ˆb > max x i Method of maximum likelihood Suppose that X is a continuous random variable with pdf f(x θ 1,..., θ k ) or a discrete rv with probability function P (X = x θ 1,..., θ k ). We make the following definition. Definition 9.3 Let X 1,..., X n be a random sample from X and let x 1,..., x n be the observed values of X 1,..., X n. The function of the parameters θ = (θ 1,..., θ k ) defined by if X is continuous, or L(θ x 1,..., x n ) = f(x 1,..., x n θ) L(θ x 1,..., x n ) = P (X 1 = x 1,..., X n = x n θ) if X is discrete, is called the likelihood function. The distinction between the likelihood function and the pdf (or prob function) is that the former is a function of the parameters θ while the latter is a function of the variable X. Example Let X have an exponential distribution with pdf f(x θ) = θ exp( θx). 6

7 Then the likelihood function is L(θ x 1,..., x n ) = f(x 1,..., x n θ) = f(x 1 θ) f(x n θ) by independence = θ exp( θx 1 ) θ exp( θx n ) = θ n exp( θ i x i ). Now consider two values of θ: θ 1 and θ 2. If L(θ 1 x 1,..., x n ) > L(θ 2 x 1,..., x n ) then the sample we have actually observed is more likely to have occurred if θ = θ 1 than if θ = θ 2. This can be interpreted as saying that θ 1 is a more plausible value for the true value θ than is θ 2. Thus it seems natural to find the value of θ for which the function L(θ x 1,..., x n ) attains a maximum. Definition 9.4 Let x 1,..., x n be a realisation of a random sample X 1,..., X n from X. We call x 1,..., x n a sample point from X. Definition 9.5 For each sample point x 1,..., x n let ˆθ be a parameter value at which L(θ x 1,..., x n ) attains its maximum. We call ˆθ a maximum likelihood estimator (MLE) of the parameter θ. Note the definition holds whether θ is a single unknown parameter or a vector of k unknown parameters. If the likelihood function is differentiable in θ i possible candidates for the MLE are the values of θ = (θ 1,..., θ k ) that solve θ i L(θ x 1,..., x n ) = 0 i = 1,..., n. Note that the zeroes of the first derivative only locate extreme points in the interior of the domain of the function. If the extreme occurs on the boundary the first derivative may not be zero. Note we should check that second derivatives are negative so that we do have a maximum. The likelihood function is a product of marginal pdf s (or probability functions). It is often useful to consider the logarithm of the likelihood function. This is called the log-likelihood function. 7

8 Example (continued) For the exponential distribution the log-likelihood is ln L(θ x 1,..., x n ) = ln[θ n exp( θ i x i )] The first derivative with respect to θ is d ln L dθ setting this equal to zero we see that ˆθ = = n ln[θ] θ x i = n θ x i n xi = 1 x. The second derivative with respect to θ for all θ. d 2 ln L dθ 2 = n θ 2 < 0 So ˆθ = 1/ x maximises ln L and so maximises L. Thus ˆθ = 1/ x is the maximum likelihood estimator of θ. Example A binomial experiment consisting of n trials resulted in observations y 1,..., y n, where y i = 1 if the ith trial was a success, y i = 0 otherwise. Find the maximum likelihood estimator of p, the probability of a success. The likelihood of the observed sample is L(p y 1,..., y n ) = p y (1 p) n y where y = y i. The log likelihood is ln L(p y 1,..., y n ) = y ln p + (n y) ln(1 p). Differentiating with respect to p we have d ln L dp = y p n y 1 p. The value of p that satisfies d ln L dp = 0 is the solution of y ˆp = n y 1 ˆp. 8

9 Solving we see that ˆp = y/n. Note that so our solution is indeed a maximum. d 2 ln L dp 2 = y p 2 n y (1 p) 2 < 0 Example Let X N(µ, 1) and let x 1,..., x n be a sample point from X. Find the MLE of µ. The likelihood is L(µ x 1,..., x n ) = = n 1 exp { 12 } (x i µ) 2 2π { 1 exp 1 (2π) n/2 2 } (x i µ) 2 and the log-likelihood is ln L(µ x 1,..., x n ) = ln(2π) n/2 + ln exp { 1 2 = n 2 ln(2π) 1 2 (x i µ) 2. The first derivative is d ln L dµ = (x i µ) = x i nµ. Setting this equal to zero we have } (x i µ) 2 The second derivative is ˆµ = 1 n xi. d 2 ln L dµ 2 = n < 0 and hence ˆµ = X is the maximum likelihood estimator. Example Let X 1,..., X n be a random sample from a uniform distribution on the interval (0, θ). The pdf of X is p(x θ) = 1 θ 0 < x < θ. 9

10 Find the maximum likelihood estimator of θ. The likelihood is given by L(θ x 1,..., x n ) = n 1 θ = 1 θ. n Note that L is a monotonically decreasing function of θ and hence that nowhere in the interval 0 < θ < is dl dθ equal to zero. However note that L increases as θ decreases and that θ must be equal to or greater than the maximum observation in the set x 1, x 2,..., x n. of θ that maximises L is the largest observation in the sample. ˆθ = X (n) = max(x 1,..., X n ). Hence the value That is The following theorem tells us how to find the MLE of any function of a parameter for which we know the MLE. Theorem 9.2 Invariance property of MLE. If ˆθ is the MLE of θ, then for any function g(θ) the MLE is g(ˆθ). Example The variance of an exponential distribution having parameter θ is 1/θ 2. Since the MLE of θ is 1/ X the MLE of the variance is X 2. We note the following without proof. Maximum likelihood estimators and their properties will be discussed in more detail in Statistical Inference. Under some general conditions MLEs are consistent estimators, both their bias and variance tend to zero as n tends to infinity. For large samples MLEs have minimum variance, no other consistent estimator has effectively smaller variance. For large samples the distribution of an MLE is approximately normal. 10